Heretofore the ability to integrate real-time three-dimensional hand interaction into software applications has been some what limited and has required a relatively high level of skill among practitioners. Complex three-dimensional worlds have not been easy to simulate and frequently such simulations where performed in a rudimentary manner as the required results would not support the massive efforts needed to understand the simulated environment, objects in that environment, or the input/output devices with interaction with the environment and objects would be made. Nor has it been practical to was time developing complex scene graphs and mapping between the several scene graphs that might typically be present. System interaction and the visual and tactile/force feedback were also in need of improvement, particularly where it was desired that visual feedback cues be synchronized with tactile and force cues, especially for had grasping interactions of virtual objects.
These and other limitations have been addressed by the inventive system, method, data structures, computer program, and computer program product of the invention.